ABSTRACT
In this paper, we present a new multimodal corpus called Biometric Russian Audio-Visual Extended MASKS (BRAVE-MASKS), which is designed to analyze voice and facial characteristics of persons wearing various masks, as well as to develop automatic systems for bimodal verification and identification of speakers. In particular, we tackle the multimodal mask type recognition task (6 classes). As a result, audio, visual and multimodal systems were developed, which showed UAR of 54.83%, 72.02% and 82.01%, respectively, on the Test set. These performances are the baseline for the BRAVE-MASKS corpus to compare the follow-up approaches with the proposed systems.
ABSTRACT
Monitoring and evaluation of the safety level of individuals is one of the most important problems of the modern world, which was forced to change due to the emergence of the COVID-19 virus. To increase the safety level of individuals, new information technologies are needed that can stop the spread of infection by minimizing the threat of outbreaks and monitor compliance with recommended measures. These technologies, in particular, include intelligent tracking systems of the presence of protective face masks. For these systems, this article proposes a new method for generating training data that combines data augmentation techniques, such as Mixup and Insert. The proposed method is tested on two datasets, namely, the MAsked FAce dataset and the Real-World Masked Face Recognition Dataset. For these datasets, values of the unweighted average recalls of 98.51% and 98.50% are obtained. In addition, the effectiveness of the proposed method is tested on images with face mask imitation on people's faces, and an automated technique is proposed for reducing type I and II errors. Using the proposed automated technique, it is possible to reduce the number of type II errors from 174 to 32 for the Real-World Masked Face Recognition Dataset, and from 40 to 14 for images with painted protective face masks.
ABSTRACT
Since 2019 all countries of the world have faced the rapid spread of the pandemic caused by the COVID-19 coronavirus infection, the fight against which continues to the present day by the world community. Despite the obvious effectiveness of personal respiratory protection equipment against coronavirus infection, many people neglect the use of protective face masks in public places. Therefore, to control and timely identify violators of public health regulations, it is necessary to apply modern information technologies that will detect protective masks on people's faces using video and audio information. The article presents an analytical review of existing and developing intelligent information technologies for bimodal analysis of the voice and facial characteristics of a masked person. There are many studies on the topic of detecting masks from video images, and a significant number of cases containing images of faces both in and without masks obtained by various methods can also be found in the public access. Research and development aimed at detecting personal respiratory protection equipment by the acoustic characteristics of human speech is still quite small, since this direction began to develop only during the pandemic caused by the COVID-19 coronavirus infection. Existing systems allow to prevent the spread of coronavirus infection by recognizing the presence/absence of masks on the face, and these systems also help in remote diagnosis of COVID-19 by detecting the first symptoms of a viral infection by acoustic characteristics. However, to date, there is a number of unresolved problems in the field of automatic diagnosis of COVID-19 and the presence/absence of masks on people's faces. First of all, this is the low accuracy of detecting masks and coronavirus infection, which does not allow for performing automatic diagnosis without the presence of experts (medical personnel). Many systems are not able to operate in real time, which makes it impossible to control and monitor the wearing of protective masks in public places. Also, most of the existing systems cannot be built into a smartphone, so that users be able to diagnose the presence of coronavirus infection anywhere. Another major problem is the collection of data from patients infected with COVID-19, as many people do not agree to distribute confidential information. © 2021 St. Petersburg Federal Research Center of the Russian Academy of Sciences. All Rights Reserved.